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College level Examination Program (CLEP) Tests were 
normed on a national basis, administering the test to nationwide 
samples of sub jects. Horms appear in the booklet, CLEP Scores: 
Interpretation and Use, and consist of the test score means for 
groups of students receiving grades of A, B, C, D, and F in the 
relevant course, the proportion of students receiving each grade, the 
sample size, and the correlation coefficient between test scores and 
earned course grades. The validity of the tests for the purpose of 
selecting creditable students from the population of test takers is 
assessed through an examination of the systematic differences in test 
means across earned grades, and the correlation coefficient between 
earned grades and test scores. This paper supports the "assertion that 
the correlation coefficient is a misleading statistic for the purpose 
of validity assessment in this context. A decision theoretic 
procedure is developed which focuses on the likelihood of errors in 
test based decisions. The decision theoretic means of validity 
assessment was applied to the data of all CLEP tests discussed in the 
norming literature. The analysis showed a dramatically incoherent 
value system displayed among the tests, with wildly fluctuating error 
likelihoods and ratios. , Further , the likelihood of each error type 
was found to be substantially greater than would be supposed based on 
an examination of the correlation coefficients between earned grades 
and test scores. (A uthor/MV) 
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A NOTE ON THE. PROBABILITY 
OF ERRORS IN DECISIONS BASED ON 
TESTS OF THE COLLEGE LEVEL EXAMINATION PROGRAM 

A review of the correlation between subject area tests of the 
College Level Examination Program and relevant course grades shows 
these correlations to exist to a moderate degree in the norming sample 
(r values range from .37 to .67) J While these correlation coefficients 
are statistically significant, and even large, when compared to the 
results of most validity studies, their size may contribute to a false 
sense of confidence in the validity of the test with regard to it's use. 

Another view of the validity of the test may be developed from a 
decision theoretic viewpoint with closer association between the analytic 
procedure and test use. Let decision D^ indicate the decision that the student has 
the skills associated with a given course, and decision D^ indicate the decision 
that the student does not have such skills. Define groups , the 
students for v/hich D-j was the appropriate decision, and , the students 
for which D2 was appropriate. 

For an appropriate test, let T represent a chosen threshold. 

The rule is adopted that a score greater than T causes decision D^ 

and a score less than T causes decision D2. Two error conditions exist. 

Error Type I: Decision D_ is made for a member of P 

1 2 

Error Type II: Decision D^ is made for a member of P^ 
If x^. is the score of subject s^. , define: 
= Probability (x. > T|s^.eP2) 
B = Probability (x- < T|s^-cP^ ) 

^College Entrance Examination Board "CLEP Scores: Interpretation 
and Use" (1973) p. 22. 
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These error probabilities are illustrated below: 




Students Passing 
Class 



Test Scale ' 

Within the framework, let be the students receiving a grade of 
^ A, B or C in a course relevant to a given CLEP subject area examination. 
P^ consists of those students receiving a D or F in such courses. 
D-| and are decisions to grant or not grant credit for the course 
via the. CLEP examination. T is the recommended threshold, the mean 
scaled score of the student receiving a grade of C in the course. 
Further, ^ is the proportion of students receiving grades of D or F 
with scores above T. 6 is the proportion of students with grades of 
A, B or C who have scores below T. 

It may be noted that it would be possible to define a Perfectly 
Val id test as one where « and B are zero or extremely near zero. 
This occurs when the distributions are disjoint. 




A Completely Invalid test could be defined as one where <^ and S 
equal .5. This occurs when the distributions are coincidental. 




Test Scale 

Having developed this approach to validity analysis, it :s possible 
to compute the magnitude of cc and B for the CLEP subject area examinations 
Information for the computation is provided by Educational Testing 
Service, In this table, the means of each within final grade group, 
the percent of the total norming sample represented by each within 
final grade group and the total number of subjects is given for each 
test. From this the number of subjects receiving each grade may be 
computed. This information may be symbolized as follows: 



Mean and Percentages for 
Within Grade Groups 



Grade 
Mean 

Number 



A 
M. 

c 

n. 



B 



n. 



C 
M 

c 

n. 



D 
M 

c 

n 



F 

n. 



•a -'b "c ^''d ^'f 
The mean CLEP score for students with grades of C or better is computed as 

n. M + n. M. + n^ 
_ a a DP c c 

"a + "b + "c 

and the mean score for students with grades less than C is computed as: 

."d ^d + "d 



"d + "f 



Ibid., p. 22. 



To compute the standard deviation of the C or better and less than 
C student group, the standard deviation of the within grade groups 
is required. Since this information is not readily available from 
Educational Testing Service, an estimate is needed, A conversation 
with Walter Shea of ETS lead to the use of the value 5 as a standard 
deviation for all within grade groups. Accordingly, 

= 25,= ) (x.. - M.)2 where j = A,B,C,0, or F and n . refers 

J C—i 'J J J 

i=l 

to the number of subjects receiving a grade of j. 
Therefore: 

n . 

^ (x. .2 - 2 M. X. . + M.2 

25 =y _y i- 2i_J_ 



i=1 



n. 
J 



n . n . 

J J 



25n = y X, ,2 - 2M, V x.. + M.2 n". 
i=l i=l 

n . 

J 

= Y X. .2 - 2M.2 n. + M.2 n. 
i=l 

"j "j 

i=l i=l 

25n. = SS. - M 2 n. or SS. = n. (M.2 + 25). The standard deviation for 
J J J J J J J 

those receiving a grade of C or better may be computed as follows: 
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1 1 ^i/-v'"p 



a 



2 = 



j=a i=1 



where N_ = n, + + n and is the 



mean of the subjects with a grade of C or better. Thus, 

SS^ + SSh + SS -(n + n. + n ) w 2 
o 2 = a b c ^ a b p 

P "a + % + "c 

n (n 2 + 25) + n^ (nj- + 25) ■^ n (n 2 + 25) - (n + n + n ) g ^ 
_ a a b • b c c a b c P 

n + n, + n 
a b c 



Similarly: 

2 nd (25 M/) -H n^ (25 + M/) - (n^ 4 nj 
°f " "d + "f 



If the threshold T is defined as M^, the mean score of C group (as 
recommended by CEEB in the booklet mentioned above), 



then - = / and p = 



■f (x)dx 



-oo 



°f 



where «(x)dx.is the standard normal distribution. 

A conipLiuer program was developed to provide the computation a: 
outlined above for each CLEP subject examination. 
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The examinations and the associated error probabilities computed 

as indicated above, -are listed below. In computing and b, it 

is assumed that the test scores are normally distributed in each population. 

CLEP SUBJECT AREA EXAMINATION 
ERROR PROBAELITIES 



Exflmi nd ti on 
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Literature. Analv<;i«; ;^nH 






Interpretation of 


.05 


.31 


Marketing, Introductory 


.14 


.31 


Medical Techno! ogy 






CI incal Chemistry 


.12 


.27 


Hematology 


.12 


.21 


Inimunohematology 


.16 


.27 


Microbiology 


.21 


.27 


Money and Banking 


.15 


.29 


Psychology, General 


.10 


.30 


Sociology, Introductory 


.13 


.24 


Statistics 


.14 


.28 


Tests and Measurements 


.21 


.25 


Trigonometry 


.12 


.26 


Western Civilization 


.05 


.30 



8 

o 

ERIC 



7 



A review of the values of « and B is somewhat disturbing. While 
the assumptions of normality and that the within grade group starxJip? 
deviation is 5 may be somewhat inappropriate, and could account for 
the large size of =^ and 6 to some degree, it seems reasonable that this 
is not the entire explanation.. From this decision theoretic viewpoint, 
therefore, it appears that the validity of the subject area examinations 
is nf concern in many subject areas. 
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